Information processing device and information processing method

ABSTRACT

The present technology relates to an information processing device, an information processing method, and a program that can more exactly indicate a position outside a display region. An outside-display-region-position designation unit designates a position outside a display region of an image display unit, and a drawing/sound control unit controls output of a sound of an AR object from a sound output unit while moving the AR objet toward the designated position. The present technology can be applied to a wearable computer, for example, a glasses-type device having a pair of image display units for a left eye and a right eye.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation application of U.S. patentapplication Ser. No. 16/392,967, filed on Apr. 24, 2019, which is acontinuation application of U.S. patent application Ser. No. 15/825,435,filed on Nov. 29, 2017, now U.S. Pat. No. 10,303,421, which is acontinuation application of U.S. patent application Ser. No. 15/031,724,filed on Apr. 23, 2016, now U.S. Pat. No. 9,851,936, which is a nationalstage entry of PCT/JP2014/078305, filed on Oct. 24, 2014, which claimspriority from prior Japanese Priority Patent Application JP 2013-229121filed in the Japan Patent Office on Nov. 5, 2013, the entire content ofwhich is hereby incorporated by reference.

TECHNICAL FIELD

The present technology relates to an information processing device, aninformation processing method, and a program, and particularly relatesto an information processing device, an information processing method,and a program which can more exactly indicate a position outside adisplay region.

BACKGROUND ART

In recent years, research on wearable computers that users can carrywhile walking have been conducted (for example, Patent Literature 1). Assuch a kind of wearable computer, a display device that can be mountedin a head like a head-mounted display (which will be referred to as anHMD) or the like is known.

In addition, a technology called augmented reality (AR) that presentsvirtual content overlaid on an object of a real space to users hasgained attention (for example, refer to Patent Literature 2). By usingthis AR technology, for example, information (AR object) of a scene thata user sees through a transmissive-type display such as an HMD can bedisplayed overlaid on a place in accordance with a current position ofthe user.

Furthermore, Patent Literature 3 discloses a technology of controllingreproduction of music sources based on a current position of a user anda direction specified according to a path to a destination.

CITATION LIST Patent Literature

Patent Literature 1: JP 2011-28763A

Patent Literature 2: JP 2013-92964A

Patent Literature 3: JP 2003-028663A

SUMMARY OF INVENTION Technical Problem

Since only a limited region can be displayed in a display such as an HMDas described above, there are cases in which it is difficult to displaynot only information of the inside of the field of view of a user butalso information of the outside of the field of view of the user in thatnarrow display region. Consequently, despite the fact that display ofinformation using an image is intuitive and explicit, a limited regioncan be displayed on a display, and thus there is a limitation ondisplaying all of the information.

In addition, in Patent Literature 3 described above, a user canrecognize left and right positions on a straight line because headphonesuse two channels for reproduction; and when stereoscopic sounds areexpressed using a head-related transfer function (HRTF), however, thereis a possibility of mistakenly recognizing front and back sides if onlya sound is used for position display. One of reasons therefor is that,when the HRTF is not the user's, a sound is heard in a different wayfrom the way the user normally hears sounds of the natural world withhis or her ears, and thus the user may not be able to catch the positionof the sound. In addition, even if the HRTF is the user's, the way ofhearing may be different owing to a characteristic of headphones or areproduction device performing the reproduction.

As described above, there is a demand for indicating information of theoutside of a display region of a display device that has a limiteddisplay region, but even if the sound reproduction method disclosed inPatent Literature 3 is used, it is not possible to exactly indicateinformation of the outside of the display region.

The present technology takes the above circumstances into consideration,and aims to more exactly indicate a position outside a display region.

Solution to Problem

According to an aspect of the present technology, an informationprocessing device includes: an image display unit configured to displayan image; a sound output unit configured to output a sound; a positiondesignation unit configured to designate a position outside a displayregion of the image display unit; and a control unit configured tocontrol output of a sound of an augmented reality (AR) object whilemoving the AR object toward the designated position.

The control unit may cause an image of the AR object to be displayedwhen the AR object passes through the display region of the imagedisplay unit. There may be a plurality of AR objects.

The control unit may cause the AR objects to move on both sides of auser when the AR objects move toward the designated position.

The control unit may cause sounds of the plurality of AR objects to beoutput at different timings.

The information processing device may further include a detection unitconfigured to detect a direction of the image display unit. The controlunit may cause the AR object to move according to the direction of theimage display unit.

The control unit may cause the AR object to move in a manner that theimage of the AR object is displayed in the display region.

The control unit may cause an output position of a sound of the ARobject to be the same as a display position of the image of the ARobject inside the display region.

The control unit may cause an output position of a sound of the ARobject to be different from a display position of the image of the ARobject inside the display region.

The information processing device may be a glasses-type device having apair of the image display units for a left eye and a right eye.

The information processing device may be an independent device or aninternal block constituting one device.

An information processing method and a program according to an aspect ofthe present technology are an information processing method and aprogram that are compatible with an information processing deviceaccording to an aspect of the present technology.

In an information processing device, information processing method, andthe program according to an aspect of the present technology, a positionoutside a display region of an image display unit is designated, andoutput of a sound of an AR object from a sound output unit is controlledwhile moving the AR object toward the designated position.

Advantageous Effects of Invention

According to an aspect of the present technology, it is possible to moreexactly indicate a position outside a display region. Note that theeffect disclosed herein is not necessarily limitative, and any effectdisclosed in the present disclosure may be exhibited.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a configuration of an embodiment of anAR system to which the present technology is applied.

FIG. 2 is a block diagram showing a detailed configuration of a controlbox and a HMD.

FIG. 3 is a block diagram showing a detailed configuration of asmartphone.

FIG. 4 is a block diagram showing a detailed configuration of an ARserver.

FIG. 5 is a block diagram showing a detailed configuration of an ARprocessing unit.

FIG. 6 is a flowchart describing a process executed by each of devicesconstituting an AR system.

FIG. 7 is a flowchart describing an AR object correspondence process 1.

FIG. 8 is a diagram showing a cylindrical coordinate system of the ARsystem.

FIG. 9 is a diagram showing a relation between a display region and anAR object in the cylindrical coordinate system.

FIG. 10 is a diagram showing a relation between a display region and asound-added AR object in the cylindrical coordinate system.

FIG. 11 is a flowchart describing a sound-added AR object correspondenceprocess 1.

FIG. 12 is a diagram showing an example of a designation of a positionof a target.

FIG. 13 is a diagram showing an example of setting a trajectory of asound-added AR object.

FIG. 14 is a diagram showing a display example of a sound object image.

FIG. 15 is a flowchart describing a sound-added AR object correspondenceprocess 2.

FIG. 16 is a diagram showing an example of a designation of a positionof a target.

FIG. 17 is a diagram showing an example of setting a trajectory of asound-added AR object.

FIG. 18 is a diagram showing a display example of a sound object image.

FIG. 19 is a flowchart describing an AR object correspondence process 2.

FIG. 20 is a diagram showing a display example of a sound object image.

FIG. 21 is a diagram showing an example of a two-direction movement ofsound objects.

FIG. 22 is a diagram showing an example of an image drawing path of anAR object.

FIG. 23 is a diagram showing the concept of VPT.

FIG. 24 is a diagram showing an example of signal processing of VPT.

FIG. 25 is a diagram for describing a first sound path of an AR object.

FIG. 26 is a diagram for describing basic sound processing.

FIG. 27 is a diagram for describing sound processing of a two-directionmovement.

FIG. 28 is a diagram for describing sound processing at an intermediateposition.

FIG. 29 is a diagram for describing sound processing of a continuoussound.

FIG. 30 is a diagram for describing a second sound path of an AR object.

FIG. 31 is a diagram for describing basic sound processing.

FIG. 32 is a diagram showing a specific operation example 1 of the ARsystem.

FIG. 33 is a diagram showing a specific operation example 2 of the ARsystem.

DESCRIPTION OF EMBODIMENTS

Embodiments of the present technology will be described below withreference to the drawings.

<Configuration of an AR System>

FIG. 1 is a block diagram showing a configuration of an embodiment of anAR system to which the present technology is applied.

The AR system 1 presents information with respect to a current positionof a user wearing a head-mounted display (HMD) 20 to the user using anaugmented reality (AR) technology. As shown in FIG. 1, the AR system 1is constituted by a control box 10, the HMD 20, a smartphone 30, and anAR server 40. The control box 10 and the HMD 20 are connected to eachother by a cable that conforms to a predetermined standard. In addition,the smartphone 30 and the AR server 40 are connected to each other via awireless network, the Internet 80, or the like.

The control box 10, which is an apparatus for controlling the HMD 20,controls operations of the HMD 20 according to manipulations of variousbuttons by the user. The HMD 20, which is an example of a wearablecomputer, is a glasses-type device having transmissive-type displays,headphones, and the like. The HMD 20 has a pair of transmissive-typedisplays for a left eye and a right eye disposed at the positions oflenses that are placed in the frame of normal glasses, and is worn onthe head of the user.

In addition, the control box 10 has a short-range wireless communicationunit, and can perform wireless communication with the smartphone 30based on a short-range wireless communication standard such as Bluetooth(registered trademark) to exchange various kinds of data. The smartphone30 has a Global Positioning System (GPS) function, and can acquire acurrent position of the user wearing the HMD 20 by receiving signalsfrom a GPS satellite 90. Then, the smartphone 30 transmits informationindicating the current position to the AR server 40 via the Internet 80to acquire AR object information of the current position. The smartphone30 transmits the AR object information to the control box 10 throughwireless communication.

Here, the AR object information includes information of coordinates,images, sounds, and the like. Coordinate information refers to, forexample, the coordinates of an AR object on a cylindrical coordinatesystem around the user wearing the HMD 20 indicating a disposed positionthereof. Image information refers to information regarding an imagedisplayed as the AR object. In addition, sound information refers toinformation regarding a sound indicating the AR object. In thedescription below, image information of an AR object will be referred toas an “image object” and sound information thereof will be referred toas a “sound object.”

The AR object information, however, at least includes an image objectand coordinate information thereof, and the information arbitrarilyincludes a sound object. Thus, among AR objects, one that includes asound object in particular will be referred to as a “sound-added ARobject.” In addition, a sound object further includes image information,and an image thereof will be referred to as a “sound object image.”

The control box 10 outputs the AR object information received from thesmartphone 30 to the HMD 20. Accordingly, for example, an image objectthat relates to an object that the user wearing the HMD 20 sees throughthe transmissive-type displays can be overlaid and displayed on theobject. In addition, a sound that corresponds to the sound object can beoutput from the headphones of the HMD 20.

The AR system 1 is configured as described above.

<Configuration Examples of the Respective Devices>

Next, configuration examples of the respective devices that constitutethe AR system 1 of FIG. 1 will be described with reference to FIGS. 2 to5.

(Detailed Configurations of the Control Box and the HMD)

FIG. 2 is a block diagram showing the detailed configurations of thecontrol box 10 and the HMD 20 of FIG. 1. As described above, the controlbox 10 and the HMD 20 are connected to each other by the cable thatconforms to the predetermined standard.

As shown in FIG. 2, a central processing unit (CPU) 101, a read onlymemory (ROM) 102, and a random access memory (RAM) 103 are connected toeach other by a bus 104 in the control box 10. The CPU 101 executes acontrol program recorded in the ROM 102 to control operations of eachunit of the control box 10. In addition, the RAM 103 includes variouskinds of data appropriately recorded therein.

An input and output interface 105 is further connected to the bus 104.The input and output interface 105 is connected to a manipulation unit106, a connection unit 107, and a wireless communication unit 108. Themanipulation unit 106 is a physical button and the like provided in thecontrol box 10, and supplies manipulation signals to the CPU 101according to manipulations of the user. The CPU 101 controls operationsof each unit of the HMD 20 according to the manipulation signals fromthe manipulation unit 106.

The connection unit 107 is connected to the HMD 20 by the cable thatconforms to the predetermined standard, and performs an exchange ofvarious kinds of data with the HMD 20 according to control of the CPU101. The wireless communication unit 108 has a short-range wirelesscommunication function, performs wireless communication with thesmartphone 30 according to control of the CPU 101 based on apredetermined short-range wireless communication standard to exchangevarious kinds of data.

In addition, as shown in FIG. 2, the HMD 20 is constituted by aconnection unit 201, a geo-magnetic sensor 203, a gyro sensor 204, anacceleration sensor 205, a display 206, headphones 207, and a cameraunit 208, and the constituent elements are connected to an input andoutput interface 202.

The geo-magnetic sensor 203 detects geomagnetism around the HMD 20. Thegyro sensor 204 detects rotation angles of the HMD 20. The accelerationsensor 205 detects gravitational acceleration of the HMD 20. Detectionresults from the geo-magnetic sensor 203, the gyro sensor 204, and theacceleration sensor 205 are supplied to the connection unit 201 via theinput and output interface 202 as sensor values.

The connection unit 201 outputs the sensor values from the geo-magneticsensor 203, the gyro sensor 204, and the acceleration sensor 205 to thecontrol box 10. Accordingly, the control box 10 can detect an attitudeor a direction of the HMD 20 using the sensor values. Note that thecontrol box 10 may acquire a current position of the user using thesensor values rather than using the GPS function based on so-calledautonomous navigation.

The display 206 includes the pair of transmissive-type displays for theleft eye and the right eye described above. The display 206 displaysvarious images according to control of the control box 10. In addition,the headphones 207 are small headphones placed at positions close to theleft and right ears of the user. The headphones 207 output varioussounds according to control of the control box 10.

The camera unit 208 is an outward-facing camera configured with asolid-state image sensor such as a complementary metal oxidesemiconductor (CMOS) image sensor, and has a function of photographingsubjects viewed through the display 206. The camera unit 208 suppliesimage data obtained by photographing a subject and performingpredetermined image processing to the connection unit 201. Theconnection unit 201 outputs the image data from the camera unit 208 tothe control box 10. Accordingly, the control box 10 can perform variouskinds of processing on the image data.

(Detailed Configuration of the Smartphone)

FIG. 3 is a block diagram showing a detailed configuration of thesmartphone 30 of FIG. 1.

As shown in FIG. 3, a CPU 301, a ROM 302, and a RAM 303 are connected toeach other by a bus 304 in the smartphone 30. The CPU 301 executes acontrol program recorded in the ROM 302 to control various operations ofthe smartphone 30. In addition, the RAM 303 has various kinds of dataappropriately recorded therein.

An input and output interface 305 is further connected to the bus 304. Atouch panel 306, a speaker 307, a GPS unit 308, a wireless communicationunit 309, and a mobile communication unit 310 are connected to the inputand output interface 305.

The touch panel 306 is constituted by a display unit 321 and a touchsensor 322 that is overlaid on the screen of the display unit. Thedisplay unit 321 is configured by a liquid crystal display (LCD) or thelike, and displays various kinds of information according to control ofthe CPU 301. In addition, the touch sensor 322 detects an inputmanipulation performed by the user on the touch panel 306 along with theposition on the touch panel 306 at which the manipulation is performed,and supplies a detection signal to the CPU 301. The CPU 301 controlsoperations of the units of the smartphone 30 according to the detectionsignal from the touch sensor 322.

The speaker 307 outputs sounds corresponding to sound signals accordingto control of the CPU 301. In addition, the GPS unit 308 acquirescurrent positions of the user by receiving signals from the GPSsatellite 90 via an antenna according to control of the CPU 301.

The wireless communication unit 309 has a short-range wirelesscommunication function, and thus performs wireless communication thatconforms to a predetermined short-range wireless communication standardwith the control box 10 according to control of the CPU 301 to exchangevarious kinds of data. In addition, the mobile communication unit 310performs communication with the AR server 40 and the like via theInternet 80 according to control of the CPU 301 to exchange variouskinds of data. Note that, although details are not illustrated, thesmartphone 30 has other functions such as a calling function like amobile telephone.

(Detailed Configuration of the AR Server)

FIG. 4 is a block diagram showing a detailed configuration of the ARserver 40 of FIG. 1.

As shown in FIG. 4, a CPU 401, a ROM 402, and a RAM 403 are connected toeach other by a bus 404 in the AR server 40. The CPU 401 executes acontrol program recorded in the ROM 402 to control various operations ofthe units of the AR server 40. In addition, the RAM 403 has variouskinds of data appropriately recorded therein.

An input and output interface 405 is further connected to the bus 404.An input unit 406, a display unit 407, a speaker 408, a communicationunit 409, an AR object retaining unit 410, and a drive 411 are connectedto the input and output interface 405.

The input unit 406 includes a keyboard, a mouse, a microphone, and thelike, and supplies input information to the CPU 401. The display unit407 is configured by a liquid crystal display or the like, and displaysvarious kinds of information according to control of the CPU 401. Inaddition, the speaker 408 outputs sounds according to control of the CPU401. The communication unit 409 performs communication with thesmartphone 30 via the Internet 80 according to control of the CPU 401 toexchange various kinds of data.

The AR object retaining unit 410 retains AR object information. The ARobject information is, for example, prepared in advance as data to beoverlaid on an object of a real space, and recorded in the AR objectretaining unit 410. The AR object retaining unit 410 supplies the ARobject information to the communication unit 409 according to control ofthe CPU 401. The communication unit 409 transmits the AR objectinformation read from the AR object retaining unit 410 to the smartphone30 via the Internet 80.

The drive 411 is for appropriately loading a removable medium such as amagnetic disk, an optical disc, a magneto-optical disc, or asemiconductor memory to drive the removable medium according to controlof the CPU 401.

(Detailed Configuration of an AR Processing Unit)

FIG. 5 is a diagram showing a detailed configuration of an AR processingunit 500. The AR processing unit 500 is realized as software when, forexample, the CPU 101 of the control box 10 executes the control program.The function of the AR processing unit 500, however, may be realized byanother electronic apparatus such as the HMD 20. In addition, thefunction of the AR processing unit 500 may be realized by, for example,an electronic apparatus with integrated functions of the control box 10,the HMD 20, and even the smartphone 30.

As shown in FIG. 5, the AR processing unit 500 includes a sensor valueacquisition unit 501, an AR object information acquisition unit 502, anHMD direction computation unit 503, a display region coordinatecomputation unit 504, a drawing processing unit 505, a sound processingunit 506, and a sound-added AR object correspondence processing unit507.

The sensor value acquisition unit 501 acquires sensor values detected bythe geo-magnetic sensor 203, the gyro sensor 204, and the accelerationsensor 205 of the HMD 20, and supplies the values to the HMD directioncomputation unit 503. In addition, the AR object information acquisitionunit 502 acquires AR object information of a current position receivedfrom the AR server 40 and supplies information included in the AR objectinformation to each of the drawing processing unit 505, the soundprocessing unit 506, and the sound-added AR object correspondenceprocessing unit 507. Specifically, coordinate information out of theinformation included in the AR object information is supplied to thedrawing processing unit 505 and the sound-added AR object correspondenceprocessing unit 507, an image object is supplied to the drawingprocessing unit 505 and the sound-added AR object correspondenceprocessing unit 507, a sound object is supplied to the sound processingunit 506 and the sound-added AR object correspondence processing unit507, and a sound object image is supplied to the sound-added AR objectcorrespondence processing unit 507.

The HMD direction computation unit 503 computes a direction of the HMD20 worn on the head of the user based on the sensor values supplied fromthe sensor value acquisition unit 501, and supplies the direction to thedisplay region coordinate computation unit 504 and the sound-added ARobject correspondence processing unit 507.

The display region coordinate computation unit 504 computes thecoordinates of a display region (which will also be referred to as a“display region A” hereinafter) of the display 206 of the HMD 20 basedon information indicating the direction of the HMD 20 supplied from theHMD direction computation unit 503, and supplies the computation resultto the drawing processing unit 505 and the sound-added AR objectcorrespondence processing unit 507 as display region information.

The drawing processing unit 505 receives the supply of the displayregion information from the display region coordinate computation unit504, the coordinate information, and the image object from the AR objectinformation acquisition unit 502. The drawing processing unit 505 causesthe image object to be displayed on the display 206 of the HMD 20 basedon the display region information and the coordinate information. Inaddition, the drawing processing unit 505 causes the sound object imageto be displayed on the display 206 of the HMD 20 according to control ofthe sound-added AR object correspondence processing unit 507.

The sound processing unit 506 causes a sound corresponding to a soundsignal of the sound object supplied from the AR object informationacquisition unit 502 to be output from the headphones 207 of the HMD 20according to control of the sound-added AR object correspondenceprocessing unit 507.

The sound-added AR object correspondence processing unit 507 performsprocessing for a sound-added AR object. The sound-added AR objectcorrespondence processing unit 507 is constituted by anoutside-display-region-position designation unit 521, a trajectorysetting unit 522, and a drawing/sound control unit 523.

The outside-display-region-position designation unit 521 designates aposition of a place desired to be indicated (hereinafter referred to asa “target T”) outside the display region A based on the display regioninformation supplied from the display region coordinate computation unit504. For example, when an AR object present outside the display region Ais set as the target T, the coordinate information of the image objectis designated as the position of the target T. The target T, however, isnot limited to an AR object, and it may be any object, for example, anobject present in a real space whose position can be designated.

The trajectory setting unit 522 sets a trajectory of the sound objectsupplied from the AR object information acquisition unit 502. However,the trajectory setting unit 522 sets a trajectory that passes throughthe display region A at least once as the trajectory of the soundobject.

The drawing/sound control unit 523 controls the drawing processing unit505 according to the trajectory set by the trajectory setting unit 522to display a sound object image on the display 206 of the HMD 20. Inaddition, the drawing/sound control unit 523 controls the soundprocessing unit 506 according to information such as the trajectory setby the trajectory setting unit 522 to output a sound corresponding tothe sound signal of the sound object from the headphones 207 of the HMD20.

In addition, the drawing/sound control unit 523 controls the soundprocessing unit 506 based on the information indicating the direction ofthe HMD 20 supplied from the HMD direction computation unit 503 to matchthe height of the AR object (the sound object) with the central heightof the display region of the display 20 of the HMD 20.

<Details of a Specific Process Executed By Respective Devices>

First, a process executed by the respective devices constituting the ARsystem 1 and the flow of information according to the processes will bedescribed with reference to the flowchart of FIG. 6. In the flowchart ofFIG. 6, however, processes from Steps S101 to S103 are executed by thecontrol box 101, processes from Steps S201 to S204 are executed by theHMD 20. In addition, processes from Steps S301 to S305 are executed bythe smartphone 30, and processes from Step S401 to S403 are executed bythe AR server 40.

As shown in FIG. 6, the HMD 20 determines whether a predeterminedinterval time, which is a timing at which each sensor performs sensing,has elapsed (S201), and when the predetermined interval time haselapsed, the process proceeds to Step S202. Then, since the geo-magneticsensor 203, the gyro sensor 204, and the acceleration sensor 205 performthe sensing, the connection unit 201 outputs the detection result of thesensors to the control box 10 as sensor values (S202). Note that the HMD20 performs a process of acquiring the sensor values, for example, 30times per second.

In the control box 10, the sensor value acquisition unit 501 controlsthe connection unit 107 to acquire the sensor values output from the HMD20 (S101).

In addition, the smartphone 30 determines whether a predeterminedinterval time, which is a timing at which the GPS satellite 90 acquiresa current position, has elapsed (S301), and when the predeterminedinterval time has elapsed, the process proceeds to Step S302 as shown inFIG. 6. Then, the GPS unit 308 receives a signal from the GPS satellite90 via an antenna according to control of the CPU 301 to acquire thecurrent position of the user (S302). The mobile communication unit 310transmits information indicating the current position of the user to theAR server 40 via the Internet 80 according to control of the CPU 301(S303).

In the AR server 40, the communication unit 409 acquires the informationindicating the current position transmitted from the smartphone 30 viathe Internet 80 according to control of the CPU 401 when the informationindicating the current position is transmitted from the smartphone 30(S401). Then, the CPU 401 acquires AR object information correspondingto the information indicating the current position acquired by thecommunication unit 409 from the AR object retaining unit 410 (S402). Thecommunication unit 409 transmits the AR object information correspondingto the information indicating the current position to the smartphone 30that is a request source via the Internet 80 according to control of theCPU 401 (S403).

In the smartphone 30, the mobile communication unit 310 receives the ARobject information transmitted from the AR server 40 via the Internet 80according to control of the CPU 301 (S304). In addition, the wirelesscommunication unit 309 performs wireless communication with the controlbox 10 based on a predetermined short-range wireless communicationstandard according to control of the CPU 301 to transmit the AR objectinformation (S305). Note that the smartphone 30 performs a process ofacquiring the AR object information corresponding to the currentposition, for example, once every 10 seconds.

In the control box 10, the AR object information acquisition unit 502controls the wireless communication unit 108 to perform wirelesscommunication with the smartphone 30 based on the predeterminedshort-range wireless communication standard, and thereby receives the ARobject information (S102). Then, the AR processing unit 500 performs anAR object correspondence process (S103) according to the AR objectinformation based on the sensor values acquired from the HMD 20 and thecurrent position of the user provided from the AR server 40. In the ARobject correspondence process, an image object of an AR object insidethe display region A of the display 206 of the HMD 20 is displayed, andan AR object outside the display region A is indicated by a soundobject. Details of the AR object correspondence process will bedescribed below with reference to the flowcharts of FIGS. 7 and 19.

Then, by performing the AR object correspondence process, an imagecorresponding to the AR object correspondence process is displayed onthe display 206 of the HMD 20 (S203), and a sound corresponding to theAR object correspondence process is output from the headphones 207 ofthe HMD 20 (S204).

The process executed by the respective devices constituting the ARsystem 1 and the flow of the information according to the processes havebeen described above.

(AR Object Correspondence Process 1)

Next, the AR object correspondence process 1 that corresponds to StepS103 of FIG. 6 executed by the AR processing unit 500 will be describedwith reference to the flowchart of FIG. 7.

In Step S501, the HMD direction computation unit 503 computes thedirection of the HMD 20 based on the sensor values from the sensor valueacquisition unit 501, and supplies the direction to the display regioncoordinate computation unit 504.

In Step S502, the display region coordinate computation unit 504computes the coordinates of the display region A of the display 206 ofthe HMD 20 based on the information indicating the direction of the HMD20 from the HMD direction computation unit 503, and supplies thecomputation result to the drawing processing unit 505 and thesound-added AR object correspondence processing unit 507 as displayregion information.

In Step S503, the drawing processing unit 505 determines whether thereis an overlapping part between the display region A of the display 206and a target image object (an AR object) based on the coordinateinformation of the image object from the AR object informationacquisition unit 502 and the display region information from the displayregion coordinate computation unit 504. If it is determined that thereis an overlapping part in Step S503, the process proceeds to Step S504,

In Step S504, the drawing processing unit 505 draws an image object ofthe overlapping part in the display region A of the display 206according to the determination result of Step S503. Here, a detailedprocedure of the drawing process of the image object (the AR object)will be described with reference to FIGS. 8 and 9.

As shown in FIG. 8, when an image object (an AR object) is to bepresented to a user 2 who is wearing the HMD 20, a cylindricalcoordinate system C is defined around the user 2 serving as the centerthereof, and positions of the display region A and an image object Obare decided on the cylindrical coordinate system C. In other words, thedisplay region A in accordance with the direction of the HMD 20 andimage objects Ob 1 and Ob 2 in accordance with the current position ofthe user are disposed in the cylindrical coordinate system C of FIG. 8.In addition, as shown in FIG. 9, the display region A and the imageobject Ob1 overlap each other in an overlap OL. Thus, in the drawingprocess of the image object of Step S504 of FIG. 7, the overlap OL ofthe image object Ob 1 is drawn in the display region A of the display206.

Returning to the flowchart of FIG. 7, if it is determined that there isno overlapping part in Step S503, the process of Step S504 is skipped,and the process proceeds to Step S505. For example, since the imageobject Ob 2 of FIG. 8 is outside the display region A, it is not atarget of the drawing process of Step S504.

In Step S505, it is determined whether a sound is added to the target ARobject. If a sound is determined to be added to the target AR object, inother words, if the object is determined to be a sound-added AR objectin Step S505, the process proceeds to Step S506.

In Step S506, the sound-added AR object correspondence processing unit507 performs the sound-added AR object correspondence process based onthe AR object information acquired by the AR object informationacquisition unit 502 and the coordinates of the display region computedby the display region coordinate computation unit 504. In thesound-added AR object correspondence process, a process for indicatingthe position of the target T outside the display region A to the user 2who is wearing the HMD 20 is performed.

In other words, when the cylindrical coordinate system C of FIG. 8 isviewed from above as shown in FIG. 10, for example, and when there is noAR object overlapping the display region A and only a sound-added ARobject Ob is present outside the display region A, a differential angleformed between the direction of a line of sight (a front direction) ofthe user wearing the HMD 20 and the direction of the sound-added ARobject Ob is computed as object direction information. Then, using thisobject direction information, the output of a sound is controlled sothat the sound of the sound object is heard from the directionindicating the sound-added AR object Ob (the target T), and thereby theposition of the sound-added AR object Ob (the target T) outside thedisplay region A is indicated. Details of the sound-added AR objectcorrespondence process will be described with reference to theflowcharts of FIGS. 11 and 15.

Returning to the flowchart of FIG. 7, when the target AR object isdetermined not to have a sound added thereto in Step S505, the processof Step S506 is skipped, and the process proceeds to Step S507. In StepS507, it is determined whether all AR objects acquired by the AR objectinformation acquisition unit 502 have been processed. If it isdetermined that all of the AR objects have not been processed in StepS507, the process proceeds to Step S508.

In Step S508, one AR object that has not yet undergone the process isselected as the next target AR object. In addition, when the processesof Steps S503 to S506 described above are repeated and then an imageobject of the selected target AR object is on the display region A, anoverlapping part thereof is drawn (S504), and when the further selectedAR object is a sound-added AR object, a sound-added AR objectcorrespondence process is executed (S506).

When the processes are executed sequentially on one or a plurality of ARobjects acquired by the AR object information acquisition unit 502 asdescribed above and it is determined that all of the AR objects havebeen processed in Step S507, the AR object correspondence process 1 ofFIG. 7 ends. The process thereafter returns to Step S103 of FIG. 6, andthe succeeding processes are executed.

The AR object correspondence process 1 has been described above. As thesound-added AR object correspondence process is performed in the ARobject correspondence process 1, a position outside the display region Aof the display 206 of the HMD 20 (for example, a position of the targetT such as a sound-added AR object) is designated, and the output of asound of a sound object is controlled while the sound-added AR objectmoves toward the designated position. As a result, the position outsidethe display region A can be more exactly indicated, and thus the usercan be aware of the position.

Note that, with respect to the AR object correspondence process 1 ofFIG. 7, the process is performed, for example, 30 times per second forone AR object.

(Sound-Added AR Object Correspondence Process 1)

Next, the sound-added AR object correspondence process 1 thatcorresponds to Step S506 of FIG. 7 executed by the sound-added AR objectcorrespondence processing unit 507 will be described with reference tothe flowchart of FIG. 11.

In Step S521, the outside-display-region-position designation unit 521determines whether there is a target T desired to be indicated outsideof the display region A. When it is determined that there is a target Tdesired to be indicated outside of the display region A in Step S521,the process proceeds to Step S522.

In Step S522, the outside-display-region-position designation unit 521designates the position of the target T outside the display region A.For example, the position of the target T (for example, a sound-added ARobject or the like) outside the display region A in the cylindricalcoordinate system C is designated as shown in FIG. 12.

Returning to the flowchart of FIG. 11, the trajectory setting unit 522sets a trajectory of a sound object in Step S523. However, thetrajectory of the sound object is desirably a trajectory that passesthrough the display region A at least once. For example, positions P1 toP7 in the cylindrical coordinate system C are set as the trajectory ofthe sound object as shown in FIG. 13. On the trajectory of FIG. 13, thesound object sequentially moves through the positions P1 to P7 in thecylindrical coordinate system C with respect to the user 2 who is at thecenter of the cylindrical coordinate system C. In FIG. 13, a part of thetrajectory from the positions P4 to P6 passes through the display regionA.

Returning to the flowchart of FIG. 11, the drawing/sound control unit523 sets a sound object image in Step S524. The sound object image is animage displayed on the display region A when a sound object passesthrough the display region A, and an image that can facilitateidentification of the target T is preferable. Thus, as the sound objectimage, for example, an arrow pointing at the target T or the like ispossible. For the sound object image, however, a fixed image may be setas default, and in that case, the process of Step S524 is omitted.

In Step S525, the drawing/sound control unit 523 controls the soundprocessing unit 506 to generate a starting point in the cylindricalcoordinate system C, i.e., a sound object at the position P1 on thetrajectory of FIG. 13. After the sound object is generated, the processproceeds to Step S526. In Step S526, the drawing/sound control unit 523controls the sound processing unit 506 according to the trajectory setby the trajectory setting unit 522 to move the sound object at aconstant speed. In Step S527, the drawing/sound control unit 523determines whether the sound object passes through the display region A.If the sound object is determined to pass through the display region Ain Step S527, the process proceeds to Step S528.

In Step S528, the drawing/sound control unit 523 controls the drawingprocessing unit 505 to display the sound object image on the display 206of the HMD 20. On the other hand, when the sound object is determinednot to pass through the display region A in Step S527, the process ofStep S528 is skipped, and the process proceeds to Step S529. In StepS529, the drawing/sound control unit 523 determines whether the soundobject has reached an end point. When the sound object is determined tohave not reached the end point in Step S529, the process returns to StepS526, and the succeeding processes are repeated.

In addition, as the processes from Steps S526 to S529 are repeated, thesound object starts the output of a sound from the position P1 thatserves as the starting point on the trajectory of FIG. 13 and moves tothe positions P2 to P7 in the cylindrical coordinate system C at aconstant speed while continuing the output of the sound, and when itreaches the position P7, the sound object stops the output of the sound.Since a partial section (for example, the section from the position P4to the position P5 or the like) on the entire trajectory is the displayregion A that the sound object passes through, a sound object image suchas an arrow is displayed in the display region A of the display 206.

If the sound object is determined to have reached the end point, i.e.,the position P7 in Step S529, the process proceeds to Step S530. In StepS530, the drawing/sound control unit 523 controls the sound processingunit 506 to erase the sound object that has reached the end point.

Specifically, as shown in FIG. 14, when an up-down direction of thedrawing is set to a time axis and the sound object is at the position P2at a time t1, the sound object Ob is outside of the display region A asshown in a cylindrical coordinate system C_(t1), and thus, only theoutput of the sound is performed and display of the sound object imageis not performed. Then, when the sound object Ob moves to the positionP5 at a time t2, the sound object Ob is positioned inside the displayregion A as shown in a cylindrical coordinate system C_(t2), and thusthe display of the sound object image is performed along with the outputof the sound. Note that, in practice, the image of the sound objectimage like an arrow moving at a constant speed and traversing thedisplay region A from left to right is displayed.

Accordingly, since the sound object image comes into the visual field ofthe user 2 and the movement speed of the sound object Ob can be seen,the user 2 can visually recognize the image of the arrow as well as thesound, and therefore can intuitively catch the presence of the target Ton the right side of the display region A.

Then, when the sound object Ob moves to the position P7 at a time t3,the sound object Ob passes through the display region A and ispositioned outside of the region as shown in a cylindrical coordinatesystem C_(t3), and only the sound is output. Then, when the sound objectOb reaches the position P7, the sound object is erased and the output ofthe sound stops as shown in a cylindrical coordinate system C_(t4) at atime t4.

Accordingly, after the user 2 recognizes the sound object image that ismoving in the display region A at the constant speed, the user perceivesthe time taken from when the sound object image leaves the visual fieldto when the sound is erased, and thereby, the user can intuitivelyperceive how far on the right side the position of the target T is fromthe direction that the user is facing.

Returning to the flowchart of FIG. 11, it is determined whether the user2 is facing the direction of the position of the target T in Step S531.Here, using the sensor values of the various sensors provided in the HMD20, for example, the direction of the face of the user 2 after the soundobject reaches the position P7 and is erased is computed, and whetherthe user 2 is facing the direction of the position of the target T isdetermined. If the user 2 is determined not to be facing the directionof the position of the target T in Step S531, the process returns toStep S525.

Then, by repeating the processes of Steps S525 to S531, the process forindicating the position of the target T outside of the display region Ais performed again using the sound of the sound object and the soundobject image, and if the user 2 is determined to be facing the directionof the position of the target T, the sound-added AR objectcorrespondence process 1 of FIG. 11 ends. In addition, when it isdetermined that there is no target T desired to be indicated outside ofthe display region A in Step S521, the sound-added AR objectcorrespondence process 1 of FIG. 11 ends. Then, the process returns toStep S506 of FIG. 7, and the succeeding processes are repeated.

The sound-added AR object correspondence process 1 has been describedabove. In the sound-added AR object correspondence process 1, in thecase in which, while a sound-added AR object moves to a designatedposition (for example, the position of the target T such as thesound-added AR object), the output of the sound of the sound object iscontrolled, and when the sound object passes through the display regionA, the sound object image thereof is displayed in the display region A.As a result, the sound and the image more exactly indicate a positionthereof outside of the display region A, and thus the user can be awareof the position.

(Sound-Added AR Object Correspondence Process 2)

Next, the sound-added AR object correspondence process 2 thatcorresponds to Step S506 of FIG. 7 executed by the sound-added AR objectcorrespondence processing unit 507 will be described with reference tothe flowchart of FIG. 15.

When there is a target T that is desired to be indicated outside of thedisplay region A on the display 206 of the HMD 20 in Steps S541 to S542,as in Steps S521 and S522 of FIG. 11, a position of the target T isdesignated. Here, a position of the target T (for example, a sound-addedAR object or the like) outside of the display region A is designated inthe cylindrical coordinate system C as shown in, for example, FIG. 16.

Returning to FIG. 15, a trajectory of a sound object and a sound objectimage are respectively set in Steps S543 and S544 as in Steps S523 andS524 of FIG. 11. Here, for example, the positions P1 to P7 of thecylindrical coordinate system C are set as the trajectory of the soundobject as shown in FIG. 17. On the trajectory of FIG. 17, the soundobject sequentially moves through the positions P1 to P5, approaches theuser 2 who is at the center of the cylindrical coordinate system C tosome degree, then sequentially moves through the positions P5 to P7, andmoves in the right direction of the user 2. In FIG. 17, in addition tothe trajectory of the positions P1 to P5, a part of the trajectory ofthe positions P5 and P6 is within the display region A. In addition, asthe sound object image, for example, “!” (an exclamation mark) or thelike can be set.

Returning to the flowchart of FIG. 15, the drawing/sound control unit523 controls the sound processing unit 506 to generate a sound object atthe starting point in the cylindrical coordinate system C, i.e., theposition P1 on the trajectory of FIG. 17 in Step S545. When the soundobject is generated, the process proceeds to Step S546.

The drawing/sound control unit 523 determines whether it is a timing ofa temporary stop of the sound object in Step S546. When it is determinedto be a timing of a temporary stop of the sound object in Step S546, theprocess proceeds to Step S547.

The drawing/sound control unit 523 controls the sound processing unit506 to temporarily stop the movement of the sound object in Step S547.Accordingly, the sound object stops at a certain position on thetrajectory. Then, it is determined whether it is a timing to terminatethe temporary stop of the sound object that temporarily stops in StepS548. When it is determined in Step S548 that it is not to be the timingto terminate the temporary stop, the determination process of Step S548is repeated. In other words, in that case, the sound object continuesstaying at a certain position on a certain trajectory.

On the other hand, when it is determined that it is the timing toterminate the temporary stop in Step S548, the process proceeds to StepS549. In addition, when it is determined in Step S546 that it is not thetiming of a temporary stop of the sound object, the processes of StepsS547 and S548 are skipped, and the process proceeds to Step S549.

In Step S549, the drawing/sound control unit 523 controls the soundprocessing unit 506 to move the sound object at a constant speed tofollow the trajectory set by the trajectory setting unit 522.

In Steps S550 and S551, it is determined whether the sound object passesthrough the display region A, and when the sound object passes throughthe display region A, a sound object image is displayed on the display206 of the HMD 20, as in Steps S527 and S528 of FIG. 11. In addition, inStep S552, the drawing/sound control unit 523 determines whether thesound object has reached the end point. When it is determined that thesound object has not reached the end point in Step S552, the processreturns to Step S546, and the succeeding processes are repeated.

In other words, by repeating the processes of Steps S546 to S552, thesound object starts the output of a sound from the position P1 servingas the starting point on the trajectory of FIG. 17, and while the outputof the sound continues, the sound object sequentially moves through thepositions P2 to P5 on a straight line at a constant speed. Then, whenreaching the position P5, the sound object temporarily stops to stay atthe position for a predetermined period of time. Then, the sound objectresumes moving, changes its traveling direction from the position P5that is the end point of the straight trajectory to the direction of thetarget T, and then moves from the positions P5 to P7, and when itreaches the position P7, the output of the sound stops.

On the entire trajectory, however, a partial section (for example, thesection from the positions P1 to P5 or the like) is in the displayregion A as viewed from the user 2, and when the sound object is in thesection, the sound object image is displayed in the display region A ofthe display 206, and further it temporarily stops at the position P5.Therefore, when the exclamation mark (“!”) is set as the sound objectimage, for example, there is a high possibility of the user noticing theexclamation mark approaching him or her, and the possibility can befurther heightened since the exclamation mark temporarily stops in thedisplay region.

If the sound object is determined to have reached the end point, i.e.,the position P7 in Step S552, the process proceeds to Step S553. In StepS553, the drawing/sound control unit 523 controls the sound processingunit 506 to erase the sound object that has reached the end point.

As shown in FIG. 18, specifically, in the case in which the up-downdirection of the drawing is set as a time axis, when the sound object isat the position P1 at a time t1, the sound object Ob is in the displayregion A as shown in a cylindrical coordinate system C_(t1), and thus,the output of a sound and the display of a sound object image areperformed. In this case, since the position P1 is farther from the user2 than the position P5, the sound object image of the exclamation markis displayed to be small.

Then, when the sound object moves to the position P5 at a time t2, thesound object Ob is in the display region A as shown in a cylindricalcoordinate system C_(t2), and thus the output of the sound and thedisplay of the sound object image are performed. In this case, since theposition P5 is closer to the user 2 than the position P1, the soundobject image of the exclamation mark is displayed to be large. In otherwords, as the sound object sequentially moves on the trajectory of thepositions P1 to P5 in practice, the sound object image of theexclamation mark displayed in the display region A is graduallyenlarged. Accordingly, the user 2 visually recognizes the sound objectimage of the exclamation mark as well as the sound, and thus canintuitively perceive that the target T is outside the display region A.

In addition, the sound object thereafter changes its traveling directionfrom the position P5 that serves as the end point of the straighttrajectory to the direction of the target T to move from the positionsP5 to P7, and a part of the section of the positions P5 and P6 isincluded in the display region A. Thus, the user 2 sees the sound objectimage of the exclamation mark that has approached him or her temporarilystop and then move to the right, and therefore can intuitively perceivethat the target T is on the right side of the display region A.

Then, when the sound object Ob moves to the position P7 at a time t3,the sound object Ob is positioned outside of the display region A asshown in a cylindrical coordinate system C_(t3), and only the sound isoutput. Then, when the sound object Ob reaches the position P7, thesound object is erased and the output of the sound stops as shown in acylindrical coordinate system C_(t4) at a time t4.

Returning the flowchart of to FIG. 15, when the sound object that hasreached the end point is erased in the process of Step S553, the processproceeds to Step S554. In Step S554, it is determined whether the user 2is facing the direction of the position of the target T as in Step S531of FIG. 11. When it is determined that the user 2 is not facing thedirection of the position of the target T in Step S554, the processreturns to Step S545.

Then, by repeating the processes of Steps S545 to S554, the process forindicating the target T outside of the display region A is performedagain using the sound of the sound object and the sound object imagethereof, and when it is determined that the user 2 is facing thedirection of the position of the target T, the sound-added AR objectcorrespondence process 2 of FIG. 15 ends. Then, the process returns toStep S506 of FIG. 7, and the succeeding processes are repeated.

The sound-added AR object correspondence process 2 has been describedabove. In the sound-added AR object correspondence process 2, in thecase in which, while a sound-added AR object moves to a designatedposition (for example, the position of the target T such as thesound-added AR object), the output of the sound of the sound object iscontrolled, and when the sound object passes through the display regionA, the sound object image thereof is displayed in the display region A.In addition, since the sound object image can temporarily stop in thedisplay region A, the position outside of the display region A is moreexactly indicated using a sound and an image, and thus the user can besurely aware of the position.

(AR Object Correspondence Process 2)

Next, the AR object correspondence process 2 that corresponds to StepS103 of FIG. 6 executed by the AR processing unit 500 will be describedwith reference to the flowchart of FIG. 19.

The direction of the HMD 20 is computed, and further the coordinates ofdisplay region coordinates A on the display 206 of the HMD 20 arecomputed in Steps S561 and S562 as in Steps S501 and S502 of FIG. 7.

In Step S563, the drawing/sound control unit 523 matches the height ofthe sound object (the AR object) with the central height of the displayregion A of the HMD 20 based on the information indicating the directionof the HMD 20 from the HMD direction computation unit 503.

As shown in FIG. 20, when the up-down direction of the drawing is set asa time axis, specifically, the sound object Ob (the AR object Ob) movestoward the target T at a time t1 as shown in a cylindrical coordinatesystem C_(t1); however, when the user 2 who is wearing the HMD 20 on hisor her head shakes his or her head up and down, for example, the centralheight of the display region A changes. Since the user 2 has his or herhead face down at a time t2, the central height of the display region Amoves in the lower direction of the drawing as shown in a cylindricalcoordinate system C_(t2), but the height of the sound object Ob ismatched with the central height of the display region A. In addition,since the user 2 has his or her head face up at a time t3, the centralheight of the display region A moves in the upper direction of thedrawing as shown in a cylindrical coordinate system C_(t3), but theheight of the sound object Ob is matched with the central height of thedisplay region A.

By performing the control as described above, the sound object Ob (theAR object Ob) is controlled to have the same height as the centralheight of the display region A of the HMD 20 at all times, and as aresult, the sound object Ob passes through the display region A at leastonce, and thus even when the user 2 shakes his or her head up and down,or the like, he or she can visually recognize a sound object image suchas an arrow.

Returning to the flowchart of FIG. 19, an image object of theoverlapping part of the display region A is drawn and, when the ARobject is a sound-added AR object, the sound-added AR objectcorrespondence process of FIG. 11 or FIG. 15 is performed in Steps S564to S569 as in Steps S503 to S508 of FIG. 7.

The AR object correspondence process 2 has been described above. Whenthe sound-added AR object correspondence process is performed in the ARobject correspondence process 2, the height of a sound object (an ARobject) is matched with the central height of the display region A ofthe display 20 of the HMD 20 in the cylindrical coordinate system C, andthus even when the user shakes his or her head or the like, a positionoutside of the display region A is more exactly indicated, and the usercan be aware of the position.

Note that, although the case in which the sound object Ob shifts up anddown in the display region A has been described in FIG. 20, the ARobject correspondence process 2 can also be applied when the soundobject Ob shifts up and down outside of the display region A. Forexample, when the user 2 who is wearing the HMD 20 on his or her headshakes the head up and down, the central height of the display region Achanges, and even when the sound object Ob moves up and down outside ofthe display region A, the height of the sound object Ob can be matchedwith the central height of the display region A.

Accordingly, even when the user 2 significantly moves his or her head upand down and thus the sound object Ob significantly deviates from thedisplay region A, for example, the sound object Ob can be caused totraverse the display region A, and the user 2 can be exactly aware ofthe sound object image.

In addition, although the case in which the height of the sound objectOb that has shifted up and down is matched with the central height ofthe display region A has been described in FIG. 20, the height of thesound object Ob can be arbitrarily set without being limited to thecentral height of the display region A as long as the sound object Ob(or the sound object image) can pass through the display region A.

Note that the AR object correspondence process 2 of FIG. 19 isperformed, for example, 30 times per second for each AR object.

(Two-Direction Movement of Sound Objects)

Although the example described above shows that movement of one soundobject is controlled to indicate a position of the target T, theposition of the target T may be indicated by controlling the movement ofa plurality of sound objects.

As shown in FIG. 21, a sound object Ob 1 and a sound object Ob 2 aregenerated as sound objects for indicating a position of the target T,and movements thereof are controlled to indicate the position of thetarget T. Specifically, when a left-right direction of the drawing isset as a time axis, the two sound objects Ob 1 and Ob 2 are positionedon the rear left side of the user 2 at a time t1. Then, at a time t2after a predetermined interval time elapses from the time t1, the soundobject Ob 1 moves to the front of the user 2, and the sound object Ob 2moves to the rear right side of the user 2. In addition, at a time t3after a predetermined interval time elapses from the time t2, the twosound objects Ob 1 and Ob 2 move near the target T on the front rightside of the user 2.

In the example of FIG. 21 as described above, the plurality of soundobjects Ob 1 and Ob 2 are generated, and while one sound object Ob 1 iskept moving clockwise toward the position of the target T and the othersound object Ob 2 is moving counterclockwise toward the position of thetarget T, the output of sounds of the sound objects is controlled.However, by assigning discrete sounds as the sound of the sound objectOb 1 and the sound of the sound object Ob 2 and differentiating timingsfor outputting the sounds, the sound can be independently reproduced. Inother words, it is known that, when the sounds of the sound object Ob 1and the sound object Ob 2 are the same, it is difficult to distinguish astereoscopic sound; however, if the frequency, tempo, genre, or the likeof music is changed, for example, it is possible to independentlyreproduce sounds of a plurality of sound objects.

Accordingly, when a plurality of sound objects move toward a position ofthe target T, the objects move to both sides of the user 2, such asfront and rear, left and right, or up and down sides of the user 2, andthus, for example, even when the user 2 moves and accordingly thedisplay region A of the HMD 20 moves in the cylindrical coordinatesystem C, a sound object image can surely pass through the displayregion A.

<Details of a Drawing Process>

(Path of AR Drawing)

FIG. 22 is a diagram showing an example of a path of drawing of an ARobject.

As shown in FIG. 22, the drawing processing unit 505 is provided withimage object processing units 551-1 to 551-n (n is an integer equal toor greater than 1), and performs a drawing process of an image objectincluded in one or a plurality of AR objects. Specifically, the imageobject processing unit 551-1 is constituted by an overlap detection unit561-1 and a drawing unit 562-1.

Among AR objects 1 to n, the overlap detection unit 561-1 receives aninput of coordinate information of an image object included in the ARobject 1 and display region information. The overlap detection unit561-1 detects an overlapping part of the display region A and the targetimage object (the AR object 1) based on the coordinate information ofthe image object and the display region information, and supplies thedetection result to the drawing unit 562-1 as overlap information.

The drawing unit 562-1 draws the image object of the overlapping part inthe display region A of the display 206 based on the overlap informationfrom the overlap detection unit 561-1.

The image object processing units 551-2 to 551-n are configured the sameas the image object processing unit 551-1. In other words, the imageobject processing units 551-2 to 551-n perform the process of drawing anoverlapping part with the display region A for each image objectincluded in each AR object. When there are a plurality of AR objects,however, a mixing unit 552 performs a mixing process, and then theresult is output to the display 206 of the HMD 20.

<Details of Sound Processing>

(Overview of VPT)

In order to realize the output of a sound of a sound object with theheadphones 207 of the HMD 20 as described above, a virtual surroundtechnology is used. In the present technology, it is preferable to useVPT (a registered trademark of Sony Corporation) as such a virtualsurround technology. In VPT (Virtualphones Technology), sounds arelocalized at the outside of the head of a user, rather than the insideof the head as shown in FIG. 23, and thus a sound field in which soundsseem to be naturally reproduced from speakers disposed at the front andthe rear sides can be created.

Specifically, in the VPT, a head-related transfer function (HRTF) ismeasured as a characteristic of a transfer from a sound source to bothears, the measured HRTF is convoluted into an input signal as shown inFIG. 24, and thereby while sounds are reproduced from headphones, thesame sound field as one generated when sounds are heard from speakersof, for example, a 5.1 channel and a 7.1 channel can be realized. Thesound processing of the present technology to which the VPT is appliedwill be described below. As the VPT, however, 7-channel surround (notusing a 0.1-channel subwoofer) is set to be applied.

(First Sound Path)

FIG. 25 is a diagram for describing a first sound path of an AR object.

As shown in FIG. 25, the sound processing unit 506 is provided withsound object processing units 571-1 to 571-n (n is an integer equal toor greater than 1), and performs sound processing on sound objectsincluded in one or a plurality of AR objects. Specifically, the soundobject processing unit 571-1 is constituted by a VPT filter computationprocessing unit 581-1 and a VPT processing unit 582-1.

The VPT filter computation processing unit 581-1 receives an input ofobject direction information of the AR object 1 among the AR objects 1to n. The object direction information represents a direction of a soundobject from the user 2 who is wearing the HMD 20, and for example,information of the differential angle of FIG. 10 can be used.

The VPT filter computation processing unit 581-1 computes a VPT filtervalue based on the object direction information of the AR object 1, andsupplies the value to the VPT processing unit 582-1. When the VPT of the7-channel surround is used, for example, phantom processing or the likeis performed on a VPT filter value to dispose closer to all positions ofsound objects, and a value that is likely to give sound information to aplurality of channels among the 7 channels is obtained. However, thecoordinates of the display region A may be reflected on the VPT filtervalue if necessary.

When being supplied with the VPT filter value from the VPT filtercomputation processing unit 581-1, the VPT processing unit 582-1 updatesthe VPT filter value. For example, because the object directioninformation is computed 30 times per second, VPT filter values arecomputed and updated according to the interval. In addition, the VPTprocessing unit 582-1 receives supply of a sound stream of sound objectsfrom the AR object information acquisition unit 502. The VPT processingunit 582-1 convolutes an HRTF into a sound stream that serves as aninput signal according to the VPT filter value. The VPT-processed soundstream is supplied to a mixing unit 572. When the VPT of the 7-channelsurround is used, for example, when HRTF coefficients of the 7 channelsare multiplied by the input signal to be convoluted thereinto, an outputof two channels can be obtained.

The sound object processing units 571-2 to 571-n are each configured thesame as the sound object processing unit 571-1. In other words, thesound object processing units 571-2 to 571-n perform the VPT processaccording to VPT filter values for each sound stream of sound objectsincluded in the respective AR objects. Accordingly, the mixing unit 572receives supply of VPT-processed sound streams of the AR objects 2 to n.The mixing unit 572 performs a mixing process on the VPT-processed soundstreams 1 to n, and outputs the result to the headphones 207 of the HMD20. The sound streams have, for example, a waveform of a way fileformat, and the waveform of the VPT-processed sound signals isreproduced from the headphones 207.

(Basic Sound Processing)

FIG. 26 is a diagram for describing basic sound processing of the firstsound path.

When no subwoofer is used in 7.1-channel surround, there are 7 positionsof speakers which are C, L, R, SL, SR, SBL, and SBR as shown in FIG. 26.In other words, in 7-channel surround, a center speaker (C) is disposedin front of the user 2 who is positioned at the center of a circle, andfront speakers (L and R) are disposed at the positions on the left andright sides that form an angle of 30° with the front. In addition,surround speakers (SL and SR) are disposed at the positions on the leftand right sides that form an angle of 75° with the front, and surroundback speakers (SBL and SBR) are further disposed at the positions on theleft and right sides that form an angle of 145° with the front. Each ofthe speakers has HRTFs for the left and right ears.

Here, a case in which a trajectory that sequentially passes through aposition Pg and a position Pf from a position Pa (a starting point) to aposition Pe (an end point) is set as a trajectory of a sound objectincluded in the AR object 1 is assumed as shown in FIG. 26. In thiscase, positions of the sound object substantially coincide with thedisposition of the speakers of the 7-channel surround, and thus thesound object moving in one direction may be assigned with the HRTFs ofthe speakers corresponding to the positions of the sound object.

Specifically, ata time t1 (a first order), the VPT filter computationprocessing unit 581-1 decides to use the HRTFs of the position Pa, i.e.,of SL-ch(a) of the 7-channel surround according to the object directioninformation, and updates the VPT filter value of the VPT processing unit582-1. However, silence is assigned to channels corresponding to thepositions Pb to Pg. The VPT processing unit 582-1 convolutes the HRTF ofthe SL-ch(a) for the left ear into the waveform of the way file format(for example, a “p” sound) input as a sound signal according to the VPTfilter value. In addition, the HRTF of the SL-ch(a) for the right ear isseparately convoluted into the sound signal in a similar manner. Sincethe sound object is only for one direction and there is one sound pathin the basic sound processing, the mixing unit 572 outputs the soundsignal VPT-processed by the VPT processing unit 582-1 to the headphones207. Accordingly, the waveform of the VPT-processed sound signal isreproduced from the headphones 207 of the HMD 20.

Next, when the object direction information indicates the position Pg,the VPT filter computation processing unit 581-1 decides to use theHRTFs of SBL-ch(g) of the 7-channel surround, and updates the VPT filtervalue at a time t2 (a second order) after a predetermined interval timeelapses from the time t1. The VPT processing unit 582-1 convolutes theHRTFs of the SBL-ch(g) into a sound signal according to the VPT filtervalue. Accordingly, the waveform of the VPT-processed sound signal towhich the HRTFs of the SBL-ch(g) are applied is reproduced from theheadphones 207 of the HMD 20.

Next, when the object direction information indicates the position Pf,the VPT filter computation processing unit 581-1 decides to use theHRTFs of SBR-ch(f) of the 7-channel surround, and updates the VPT filtervalue at a time t3 (a third order) after a predetermined interval timeelapses from the time t2. The VPT processing unit 582-1 convolutes theHRTFs of the SBR-ch(f) into a sound signal according to the VPT filtervalue. Accordingly, the waveform of the VPT-processed sound signal towhich the HRTFs of the SBR-ch(f) are applied is reproduced from theheadphones 207 of the HMD 20.

Subsequently, when the object direction information indicates theposition Pe, the VPT filter computation processing unit 581-1 decides touse the HRTFs of SR-ch(e) of the 7-channel surround, and updates the VPTfilter value at a time t4 (a fourth order) after a predeterminedinterval time elapses from the time t3. The VPT processing unit 582-1convolutes the HRTFs of the SR-ch(e) into a sound signal according tothe VPT filter value. Accordingly, the waveform of the VPT-processedsound signal to which the HRTFs of the SR-ch(e) are applied isreproduced from the headphones 207 of the HMD 20.

As the series of processes are performed described above, the soundreproduced from the headphones 207 of the HMD 20 can be heard as if the“p” sound sequentially moves through the stereoscopic positions of theposition Pa, the position Pg, the position Pf, and the position Peregardless of the fact that the 2-channel headphones are being used.Note that the “p” sound given as sound information of the sound objectis an example, and another single sound may be used.

(Sound Processing of Two-Direction Movement)

FIG. 27 is a diagram for describing sound processing of two-directionmovement on the first sound path.

As shown in FIG. 27, a case in which a first trajectory thatsequentially passes through the position Pb and the position Pd from theposition Pa (the starting point) to the position Pe (the end point) isset as a trajectory of a sound object 1 included in an AR object 1 and asecond trajectory that sequentially passes through the position Pg andthe position Pf from the position Pa (the starting point) to theposition Pe (the end point) is set as a trajectory of a sound object 2included in an AR object 2 is assumed. In this case, the positions ofthe sound objects 1 and 2 on the first trajectory and the secondtrajectory substantially coincide with the disposition of the speakersof the 7-channel surround, and thus the two sound objects 1 and 2 movingin two directions may be assigned with the HRTFs of the speakerscorresponding to the positions of the sound objects. An output timing ofa sound on the first trajectory and an output timing of a sound on thesecond trajectory are set to be different.

Specifically, when object direction information indicates the positionPa that serves as the starting point of the first trajectory and thesecond trajectory, the VPT filter computation processing units 581-1 and581-2 decide to use the HRTFs of the SL-ch(a) of the 7-channel surround,and updates the VPT filter value at a time t1 (a first order). However,silence is assigned to the channels corresponding to the positions Pb toPg. The VPT processing units 582-1 and 582-2 convolute the HRTF of theSL-ch(a) for the left ear into a sound signal according to the VPTfilter value. In addition, the HRTF of the SL-ch(a) for the right ear isseparately convoluted into another sound signal in a similar manner. Inthis example of sound processing of the two-direction movement, thereare two sound paths of a sound object, and thus the mixing unit 572selects one sound signal among the sound signals VPT-processed by theVPT processing units 582-1 and 582-2 in an alternating manner, andoutputs the sound signal to the headphones 207 of the HMD 20.Accordingly, for example, the waveform of the VPT-processed sound signalis reproduced from the headphones 207 of the HMD 20.

Next, when the object direction information indicates the position Pb,the VPT filter computation processing unit 581-1 decides to use theHRTFs of L-ch(b) of the 7-channel surround, and updates the VPT filtervalue at a time t2 (a second order) after a predetermined interval timeelapses from the time t1. The VPT processing unit 582-1 convolutes theHRTFs of the L-ch(b) into a sound signal according to the VPT filtervalue. Accordingly, the waveform of the VPT-processed sound signal towhich the HRTFs of the L-ch(b) are applied is reproduced from theheadphones 207 of the HMD 20.

Next, when the object direction information indicates the position Pg,the VPT filter computation processing unit 581-2 decides to use theHRTFs of SBL-ch(g) of the 7-channel surround, and updates the VPT filtervalue at a time t3 (a third order) after a predetermined interval timeelapses from the time t1. The VPT processing unit 582-2 convolutes theHRTFs of the SBL-ch(g) into a sound signal according to the VPT filtervalue. Accordingly, the waveform of the VPT-processed sound signal towhich the HRTFs of the SBL-ch(g) are applied is reproduced from theheadphones 207 of the HMD 20.

Next, when the object direction information indicates the position Pd,the VPT filter computation processing unit 581-1 decides to use theHRTFs of R-ch(d) of the 7-channel surround, and updates the VPT filtervalue at a time t4 (a fourth order) after a predetermined interval timeelapses from the time t2. The VPT processing unit 582-1 convolutes theHRTFs of the R-ch(d) into a sound signal according to the VPT filtervalue. Accordingly, the waveform of the VPT-processed sound signal towhich the HRTFs of the R-ch(d) are applied is reproduced from theheadphones 207 of the HMD 20.

Next, when the object direction information indicates the position Pf,the VPT filter computation processing unit 581-2 decides to use theHRTFs of SBR-ch(f) of the 7-channel surround, and updates the VPT filtervalue at a time t5 (a fifth order) after a predetermined interval timeelapses from the time t3. The VPT processing unit 582-2 convolutes theHRTFs of the SBR-ch(f) into a sound signal according to the VPT filtervalue. Accordingly, the waveform of the VPT-processed sound signal towhich the HRTFs of the SBR-ch(f) are applied is reproduced from theheadphones 207 of the HMD 20.

Subsequently, when the object direction information indicates theposition Pe, the VPT filter computation processing units 581-1 and 581-2decides to use the HRTFs of SR-ch(e) of the 7-channel surround, andupdates the VPT filter value at a time t6 (a sixth order) after apredetermined interval time elapses from the times t4 and t5. The VPTprocessing unit 582-2 convolutes the HRTFs of the SR-ch(e) into a soundsignal according to the VPT filter value. Accordingly, the waveform ofthe VPT-processed sound signal to which the HRTFs of the SR-ch(e) areapplied is reproduced from the headphones 207 of the HMD 20.

As the series of processes are performed as described above, the soundreproduced (output) from the headphones 207 of the HMD 20 can be heardas if the “p” sound sequentially moves through the stereoscopicpositions of the position Pa, the position Pb, the position Pd, and theposition Pe, and the “p” sound sequentially moves through thestereoscopic positions of the position Pa, the position Pg, the positionPf, and the position Pe regardless of the fact that the 2-channelheadphones are being used. Note that the “p” sound given as soundinformation of the sound object is an example, and another single soundmay be used.

(Sound Processing of an Intermediate Position)

FIG. 28 is a diagram for describing sound processing at an intermediateposition on the first sound path.

As shown in FIG. 28, a case in which a trajectory that sequentiallypasses through the position Pg and the position Pf from the position Pa(the starting point) to the position Pe (the end point) is set as atrajectory of a sound object is assumed. In this case, it is possible tocontinuously move a sound object by causing a sound to be output at anintermediate position between the positions when, for example, soundinformation is a continuous sound like “p,” in comparison to scatteredpositions such as the position Pa, the position Pg, the position Pf, andthe position Pe.

Specifically, when object direction information indicates the positionPa that serves as the starting point of the trajectory, the VPT filtercomputation processing unit 581-1 decides to use the HRTFs of theSL-ch(a) of the 7-channel surround, and updates the VPT filter value ata time t1 (a first order). However, silence is assigned to the channelscorresponding to the positions Pb to Pg. The VPT processing unit 582-1convolutes the HRTF of the SL-ch(a) for the left ear into a sound signalaccording to the VPT filter value. In addition, the HRTF of the SL-ch(a)for the right ear is separately convoluted into the sound signallikewise. In this example of sound processing at an intermediateposition, a sound object is only for one direction and there is onesound path, and thus the mixing unit 572 outputs the sound signalVPT-processed by the VPT processing unit 582-1 to the headphones 207 ofthe HMD 20. Accordingly, for example, the waveform of the VPT-processedsound signal is reproduced from the headphones 207 of the HMD 20.

Next, when the object direction information indicates a position Px thatis an intermediate position between the position Pa and the position Pg,the VPT filter computation processing unit 581-1 decides to use theHRTFs of the SL-ch(a) and the HRTFs of the SL-ch(g) of the 7-channelsurround, and updates the VPT filter value at a time t2 (a second order)after a predetermined interval time elapses from the time t1. However,silence is assigned to the channels corresponding to the positions Pb toPf. The VPT processing unit 582-1 convolutes the HRTF of the SL-ch(a)for the left ear into the waveform of the way file format (for example,a “p” sound) input as a sound signal according to the VPT filter value.In addition, the HRTF of the SL-ch(a) for the right ear is separatelyconvoluted into the sound signal in a similar manner. Further, the VPTprocessing unit 582-1 convolutes the HRTF of the SL-ch(g) for the leftear into the waveform of the way file format (for example, the “p”sound) input as a sound signal according to the VPT filter value. Inaddition, the HRTF of the SL-ch(g) for the right ear is separatelyconvoluted into the sound signal in a similar manner.

However, although the position Px is gain of both the SL-ch(a) and theSL-ch(g), it is an intermediate position between the position Pa and theposition Pg, and thus the same gain is given. For example, the value ofthe SL-ch(a) is multiplied by 0.7 and the value of the SL-ch(g) ismultiplied by 0.7, and the output obtained by a combination thereof canbe equal to a single output from the position Pa. Accordingly, thewaveform of the VPT-processed sound signals to which the HRTFs of theSL-ch(a) and the SL-ch(g) are applied is reproduced from the headphones207 of the HMD 20, and thus the “p” sound can also be heard from theposition Px that is an intermediate position between the position Pa andthe position Pg.

Thereafter, the VPT process is performed at the position Pg, theposition Pf, and the position Pe in the same manner as the basic soundprocessing of FIG. 26, and with regard to the sound processing at anintermediate position, the same VPT process as at the position Px thatis an intermediate position is also performed at an intermediateposition between the position Pg and the position Pf and an intermediateposition between the position Pf and the position Pe, and thus the “p”sound can also be heard from these intermediate positions.

As the series of processes are performed as described above, a soundreproduced (output) from the headphones 207 of the HMD 20 can be heardas if were sequentially moving not only to the stereoscopic positions ofthe position Pa, the position Pg, the position Pf, and the position Pebut also to the intermediate positions of the above positions, and thusthe “p” sound continuously moves, regardless of the fact that the2-channel headphones are being used.

(Sound Processing of a Continuous Sound)

FIG. 29 is a diagram for describing sound processing of a continuoussound on the first sound path.

As shown in FIG. 29, a case in which a trajectory that sequentiallypasses through the position Pg and the position Pf from the position Pa(the starting point) to the position Pe (the end point) is set as atrajectory of a sound object is assumed. In this case, it is possible tocontinuously move a sound object by causing a sound to be output at eachsection between the positions when, for example, sound information is acontinuous sound like “p,” in comparison to scattered positions such asthe position Pa, the position Pg, the position Pf, and the position Pe.

Specifically, when object direction information indicates the positionPa that serves as the starting point of the trajectory, the VPT filtercomputation processing unit 581-1 decides to use the HRTFs of theSL-ch(a) of the 7-channel surround, and updates the VPT filter value ata time t1 (a first order). However, silence is assigned to the channelscorresponding to the positions Pb to Pg. The VPT processing unit 582-1convolutes the HRTF of the SL-ch(a) for the left ear into a sound signalaccording to the VPT filter value. In addition, the HRTF of the SL-ch(a)for the right ear is separately convoluted into the sound signallikewise. In this example of an intermediate position, a sound object isonly for one direction and there is one sound path, and thus the mixingunit 572 outputs the sound signal VPT-processed by the VPT processingunit 582-1 to the headphones 207 of the HMD 20. Accordingly, forexample, the waveform of the VPT-processed sound signal is reproducedfrom the headphones 207 of the HMD 20.

Next, when the object direction information indicates the position Pg,the VPT filter computation processing unit 581-1 decides to use theHRTFs of the SL-ch(a) and the SBL-ch(g) of the 7-channel surround, andupdates the VPT filter value at a time t2 (a second order) after apredetermined interval time elapses from the time t1. However, silenceis assigned to the channels corresponding to the positions Pb to Pf. TheVPT processing unit 582-1 convolutes the HRTF of the SL-ch(a) for theleft ear into the waveform of the way file format (for example, a “p”sound) input as a sound signal according to the VPT filter value.Further, the VPT processing unit 582-1 convolutes the HRTF of theSBL-ch(g) for the left ear into the waveform of the way file format (forexample, the “p” sound) input as a sound signal according to the VPTfilter value. In addition, the HRTF of the SL-ch(a) for the right earand the HRTF of the SBL-ch(g) for the right ear are separatelyconvoluted into the sound signals likewise.

Here, with regard to the gain ratio of both the SL-ch(a) and theSBL-ch(g) at the positions Pa to Pg, while the level of the SL-ch(a) isshifted from the maximum value to the minimum value, for example, thelevel of the SBL-ch(g) is shifted from the minimum value to the maximumvalue, and thereby their gain ratios are changed. Accordingly, thewaveform of the VPT-processed sound signals is reproduced from theheadphones 207 of the HMD 20, and the continuous sound “p” can be heardin the section of the positions Pa to Pg.

Thereafter, although the VPT process is performed at the position Pg,the position Pf, and the position Pe in the same manner as the basicsound processing of FIG. 26, with regard to sound processing of acontinuous sound, the same VPT process performed in the section of thepositions Pa to Pg is performed in each section of the above positions,and thus the continuous sound “p” can be heard not only in the sectionof positions Pg to Pf but the section of the positions Pf to Pe. Inother words, in all sections from the starting point to the end point,the “p” sound can be heard as if it were continuously moving.

Note that, although a continuous sound is output in the section of thepositions Pa to Pg using channel information of the position Pa that isone before the position Pg in the sound processing of a continuous sounddescribed above, for example, if channel information of the nextposition Pf can be obtained, the information may be used to output acontinuous sound in the section of the positions Pa to Pg. Accordingly,a sound can be heard by the user 2 more satisfactorily as if it werecontinuously moving.

(Second Sound Path)

FIG. 30 is a diagram for describing a second sound path of an AR object.As shown in FIG. 30, the sound processing unit 506 includes acorresponding channel control unit 591, an HRTF processing unit 592, anda dual-channeling unit 593.

The corresponding channel control unit 591 receives input of informationof AR objects 1 to n (n is an integer equal to or greater than 1). Whenreceiving input of information of a plurality of AR objects, thecorresponding channel control unit 591 generates a mixed sound anddisposes the sound on a predetermined channel according to theinformation of each of the AR objects. In addition, the correspondingchannel control unit 591 decides an HRTF to be used, and supplies theHRTF to the HRTF processing unit 592.

The HRTF processing unit 592 convolutes the HRTF into the waveform of asound signal according to the HRTF supplied from the correspondingchannel control unit 591. The dual-channeling unit 593 performsdual-channeling on the sound signal from the HRTF processing unit 592according to the number of final output channels, and outputs the resultto the headphones 207 of the HMD 20.

Note that the corresponding channel control unit 591 may have a file(for example, a file in the way format) that moves channels prepared inadvance, cause the file to be reproduced at a predetermined timing, thencause the HRTF processing unit 592 to convolute the HRTF of each of thechannels thereinto, and cause the dual-channeling unit 593 to performdual-channeling on the result.

(Basic Sound Processing)

FIG. 31 is a diagram for describing basic sound processing of the secondsound path.

A case in which a trajectory that sequentially passes through a positionPg and a position Pf from a position Pa (a starting point) to a positionPe (an end point) is set as a trajectory of a sound object is assumed asshown in FIG. 31. In this case, positions of the sound objectsubstantially coincide with the disposition of the speakers of the7-channel surround, and thus the sound object moving in one directionmay be assigned with the HRTFs of the speakers corresponding to thepositions of the sound object.

Specifically, the corresponding channel control unit 591 decides to usethe HRTFs of the position Pa, i.e., the SL-ch(a) of the 7-channelsurround according to object direction information and assigns silenceto the channels corresponding to other positions Pb to Pg at a time t1(a first order). In this example, however, there is one AR object (onlyin one direction), and thus nothing is further supplemented. Then, theHRTF processing unit 592 convolutes the HRTF of the SL-ch(a) for theleft ear into the waveform of the way file format (for example, a “p”sound) input as a sound signal. In addition, the HRTF of the SL-ch(a)for the right ear is separately convoluted into the sound signal in asimilar manner. Then, the dual-channeling unit 593 performsdual-channeling on the sound signal having 7 channels according to thenumber of final output channels, and outputs the result to theheadphones 207 of the HMD 20. Accordingly, the waveform of theVPT-processed sound signal is reproduced from the headphones 207 of theHMD 20.

Next, when the object direction information indicates the position Pg,the corresponding channel control unit 591 decides to use the HRTFs ofthe SBL-ch(g) of 7-channel surround at a time t2 (a second order) aftera predetermined interval time elapses from the time t1. Then, the HRTFprocessing unit 592 convolutes the HRTF of the SBL-ch(g) for the leftear into the waveform of the way file format input as a sound signal. Inaddition, the HRTF of the SBL-ch(g) for the right ear is separatelyconvoluted into the sound signal in a similar manner. Then, thedual-channeling unit 593 performs dual-channeling on the sound signalhaving 7 channels, and thereby the waveform of the VPT-processed soundsignals is reproduced from the headphones 207 of the HMD 20.

Next, when the object direction information indicates the position Pf,the corresponding channel control unit 591 decides to use the HRTFs ofthe SBR-ch(f) of 7-channel surround at a time t3 (a third order) after apredetermined interval time elapses from the time t2. Then, the HRTFprocessing unit 592 convolutes the HRTF of the SBR-ch(f) for the leftear into the waveform of the way file format input as a sound signal. Inaddition, the HRTF of the SBR-ch(f) for the right ear is separatelyconvoluted into the sound signal in a similar manner. Then, thedual-channeling unit 593 performs dual-channeling on the sound signalhaving 7 channels, and thereby the waveform of the VPT-processed soundsignals is reproduced from the headphones 207 of the HMD 20.

Subsequently, when the object direction information indicates theposition Pe, the corresponding channel control unit 591 decides to usethe HRTFs of the SR-ch(e) of 7-channel surround at a time t4 (a fourthorder) after a predetermined interval time elapses from the time t3.Then, the HRTF processing unit 592 convolutes the HRTF of the SR-ch(e)for the left ear into the waveform of the way file format input as asound signal. In addition, the HRTF of the SR-ch(e) for the right ear isseparately convoluted into the sound signal in a similar manner. Then,the dual-channeling unit 593 performs dual-channeling on the soundsignal having 7 channels, and thereby the waveform of the VPT-processedsound signals is reproduced from the headphones 207 of the HMD 20.

As the series of processes are performed as described above, the soundreproduced (output) from the headphones 207 of the HMD 20 can be heardas if the “p” sound were sequentially moving to the stereoscopicpositions of the position Pa, the position Pg, the position Pf, and theposition Pe regardless of the fact that the 2-channel headphones arebeing used.

MODIFIED EXAMPLE

Although the examples in which a sound object moves at a constant speedhave been described with regard to the sound processing described above,a sound object may not necessarily move at a constant speed. In otherwords, a movement speed of a sound object is changeable as long as theuser 2 can satisfactorily hear movement of a sound.

In addition, although the “p” sound given as sound information is anexample and another single sound can be used as described above, it isnot limited to any single sound, and other combined sounds may befurther used. In short, any sound can be used as long as the user 2 cansatisfactorily hear movement of the sound.

Furthermore, in the sound-added AR object correspondence process 2 ofFIG. 15, the trajectory shown in FIG. 17 has been described as being setas a trajectory of a sound object in the process of Step S543. On thetrajectory of FIG. 17, the sound object sequentially moves through thepositions P1 to P5 rather than in the left-right direction, which ismovement from a distant place to a nearby place in the cylindricalcoordinate system C. In that case, although the sound object does notmove in the horizontal direction, the magnitude of the sound or the likecan be changed according to a position through sound processing.

In FIG. 17, for example, if the volume of the position P1 that is adistant place is set to 1.0, the volume of the position P2 can be set to1.1 times the volume of the position P1, the volume of the position P3can be set to 1.2 times the volume of the position P1, the volume of theposition P4 can be set to 1.3 times the volume of the position P1, andthe volume of the position P5 can be set to 1.4 times the volume of theposition P1. In addition, in FIG. 17, with regard to an HRTFreverberation amount (FIG. 24) at the position P1 that is a distantplace, for example, the reverberation amount may set to be graduallydecrease each time a sound approaches the positions P2, P3, P4, and P5.By performing such sound processing, it is possible to give, forexample, an impression of an approaching sound to the user 2. Note thatthe sound processing is an example, and another type of sound processingcan be applied as long as the user 2 can satisfactorily hear movement ofa sound through the sound processing.

In addition, although the configuration in which a sound object moves inthe horizontal direction in the 7-channel surround of theabove-described VPT has been described, a configuration in which a soundobject moves not only in the horizontal direction but also in thevertical direction can be employed by adopting a VPT configuration in anupper direction, for example, by disposing front-high speakers (L (FH)and R (FH)) at positions immediately above front speakers (L and R).Furthermore, a more stereoscopic sound object can be configured byadopting a VPT configuration in which multiple channels more than the 7channels described above are employed.

When, for example, the front-high speakers are disposed to cause a soundobject to move in the vertical direction, and thus the sound objectmoves to an upper place and a lower place outside the display region Aas the user 2 shakes his or her head up and down, and control ofmatching the height of the sound object with the central height of thedisplay region A of the HMD 20 or the like is performed as shown in FIG.20.

In the example of FIG. 20 described above, both the sound object and thesound object image thereof move close to the center of the displayregion A, but only the sound object image may move into the displayregion A while the height of the sound object is maintained. In otherwords, while the sound object is separated from the sound object imagethereof and sound processing is performed to move the sound object to anupper place or a lower place outside the display region A, the soundobject image may be set to pass through the display region A accordingto the movement of the sound object. At this time, as the sound objectimage is traversing the display region A, for example, the sound objectimage itself such as an arrow may be displayed; however, by displaying ashadow of the arrow or the like according to the height of the soundobject, the user 2 can more stereoscopically catch movement of thesound.

An output position of the sound of a sound object may be the same as adisplay position of the sound object image passing through the displayregion A as described above, or may be different therefrom.

SPECIFIC OPERATION EXAMPLES

Next, specific operation examples of the AR system 1 of FIG. 1 will bedescribed with reference to FIGS. 32 and 33.

As an operation example 1, a case in which the AR system 1 provides anapplication for a map search is shown in FIG. 32. For example, when aninstruction of indicating the north is given by the application for themap search, the north is designated as a position of a target T. Then,as shown in the drawing in which the cylindrical coordinate system C isviewed from the top on the right side of the drawing, sound objects Ob 1and Ob 2 respectively move on both sides of the user 2 toward theposition of the target T (in the north) from a predetermined startingpoint, and then reach close to the position of the target T. As shown ina screen on the left side of the drawing, when a sound is outputaccording to a sound object, and the sound object Ob 1 passes throughthe display region A, a sound object image formed by arrows isdisplayed.

Since the sound to guide, for example, to the north is output and theimage of the arrows indicating the north traverses the display region Aas described above, more intuitive position information can be providedto the user 2.

FIG. 33 shows, as an operation example 2, a case in which the AR system1 provides a social networking service (SNS) application. When anotheruser tweets near a current position of the user 2 during activation ofthe SNS application, for example, the position at which the other userhas tweeted is specified as the position of a target T. Then, along witha sound of a sound object Ob, a notification icon that serves as a soundobject image thereof is displayed in the display region A at a time t1.Then, at times t2 and t3 after predetermined interval times elapse fromthe time t1, the sound object Ob moves toward the position of the tweetwhile the sound is being output, but when the sound object moves intothe display region A, the sound object image formed by the notificationicon is displayed.

In the same manner as described above, when another user who isparticipating in an event tweets about the event, for example, the placeof the tweet is indicated by a sound of an AR object and a notificationicon, and therefore, the user 2 can immediately move to the place toparticipate in the event.

<Description of a Computer to which the Present Disclosure is Applied>

The series of processes described above can be executed by hardware butcan also be executed by software. When the series of processes isexecuted by software, a program that constructs such software isinstalled into a computer. Here, the expression “computer” includes acomputer in which dedicated hardware is incorporated and ageneral-purpose personal computer or the like that is capable ofexecuting various functions when various programs are installed.

As one example, the program executed by the computer (CPU) may beprovided by being recorded on the removable medium as a packaged mediumor the like. The program can also be provided via a wired or wirelesstransfer medium, such as a local area network, the Internet, or adigital satellite broadcast.

In the computer, by loading the removable medium into the drive, theprogram can be installed into the recording unit via the input/outputinterface. It is also possible to receive the program from a wired orwireless transfer medium using the communication unit and install theprogram into the recording unit. As another alternative, the program canbe installed in advance into the ROM or the recording unit.

Note that the program executed by the computer may be a program in whichprocesses are carried out in a time series in the order described inthis specification or may be a program in which processes are carriedout in parallel or at necessary timing, such as when the processes arecalled.

A processing step herein for describing a program which causes thecomputer to perform various processing does not necessarily have to beprocessed chronologically in the order described in a flow chart. Italso includes processing performed in parallel or individually (forexample, parallel processing or processing by an object).

The program may be a program executed by a single computer or may be aprogram executed in a distribution manner by a plurality of computers.The program may be transmitted remotely to a computer to be executed.

Further, in the present disclosure, a system has the meaning of a set ofa plurality of configured elements (such as an apparatus or a module(part)), and does not take into account whether or not all theconfigured elements are in the same casing. Therefore, the system may beeither a plurality of apparatuses, stored in separate casings andconnected through a network, or a plurality of modules within a singlecasing.

An embodiment of the disclosure is not limited to the embodimentsdescribed above, and various changes and modifications may be madewithout departing from the scope of the disclosure. For example, thepresent disclosure can adopt a configuration of cloud computing whichprocesses by allocating and connecting one function by a plurality ofapparatuses through a network.

Further, each step described by the above-mentioned flow charts can beexecuted by one apparatus or by allocating a plurality of apparatuses.In addition, in the case where a plurality of processes are included inone step, the plurality of processes included in this one step can beexecuted by one apparatus or by sharing a plurality of apparatuses.

Additionally, the present technology may also be configured as below.

(1)

An information processing device including:

an image display unit configured to display an image;

a sound output unit configured to output a sound;

a position designation unit configured to designate a position outside adisplay region of the image display unit; and

a control unit configured to control output of a sound of an augmentedreality (AR) object while moving the AR object toward the designatedposition.

(2)

The information processing device according to (1), wherein the controlunit causes an image of the AR object to be displayed when the AR objectpasses through the display region of the image display unit.

(3)

The information processing device according to (1) or (2), wherein thereare a plurality of AR objects.

(4)

The information processing device according to (3), wherein the controlunit causes the AR objects to move on both sides of a user when the ARobjects move toward the designated position.

(5)

The information processing device according to (3) or (4), wherein thecontrol unit causes sounds of the plurality of AR objects to be outputat different timings.

(6)

The information processing device according to (1) or (2), furtherincluding:

a detection unit configured to detect a direction of the image displayunit,

wherein the control unit causes the AR object to move according to thedirection of the image display unit.

(7)

The information processing device according to (6), wherein the controlunit causes the AR object to move in a manner that the image of the ARobject is displayed in the display region.

(8)

The information processing device according to (7), wherein the controlunit causes an output position of a sound of the AR object to be thesame as a display position of the image of the AR object inside thedisplay region.

(9)

The information processing device according to (7), wherein the controlunit causes an output position of a sound of the AR object to bedifferent from a display position of the image of the AR object insidethe display region.

(10)

The information processing device according to (1), which is aglasses-type device having a pair of the image display units for a lefteye and a right eye.

(11)

An information processing method of an information processing device,the method causing the information processing device to perform:

designating a position outside a display region of an image displayunit; and

controlling output of a sound of an AR object from a sound output unitwhile moving the AR object toward the designated position.

(12)

A program causing a computer to function as:

a position designation unit configured to designate a position outside adisplay region of an image display unit; and

a control unit configured to control output of a sound of an AR objectfrom a sound output unit while moving the AR object toward thedesignated position.

REFERENCE SIGNS LIST

-   1 AR system-   10 control box-   20 HMD-   30 smartphone-   40 AR server-   101 CPU-   107 connection unit-   108 wireless communication unit-   201 connection unit-   203 geo-magnetic sensor-   204 gyro sensor-   205 acceleration sensor-   206 display-   207 headphones-   301 CPU-   309 wireless communication unit-   310 mobile communication unit-   401 CPU-   409 communication unit-   410 AR object retaining unit-   500 AR processing unit-   501 sensor value acquisition unit-   502 AR object information acquisition unit-   503 HMD direction computation unit-   504 display region coordinate computation unit-   505 drawing processing unit-   506 sound processing unit-   507 sound-added AR object handing processing unit-   521 outside-display-region-position designation unit-   522 trajectory setting unit-   523 drawing/sound control unit

What is claimed is:
 1. An information processing device, comprising: acentral processing unit (CPU) configured to: designate a first positionoutside of a display region of an image display screen; set a trajectoryof an augmented reality (AR) object, wherein the trajectory of the ARobject is set as a plurality of positions; and control the AR object,based on a second position of the display region, a third position ofthe AR object, a user's position, movement of the AR object towards thedesignated first position, and a height of the AR object, wherein the ARobject is passable through the display region of the image displayscreen.
 2. The information processing device according to claim 1,wherein the CPU is further configured to match the height of the ARobject with a central height of the display region.
 3. The informationprocessing device according to claim 1, wherein the informationprocessing device is a glasses-type device having a pair of imagedisplay screens for a left eye and a right eye.
 4. The informationprocessing device according to claim 1, wherein the CPU is furtherconfigured to arbitrarily set the height of the AR object independent ofa central height of the display region.
 5. The information processingdevice according to claim 1, wherein the CPU is further configured todisplay an image of the AR object based on an overlap between the ARobject and the display region of the image display screen.
 6. Theinformation processing device according to claim 1, wherein the CPU isfurther configured to display the AR object on the display region toindicate a direction of a target object to a user.
 7. The informationprocessing device according to claim 1, wherein the CPU is furtherconfigured to control the movement of the AR object such that, an imageof the AR object is displayed in the display region.
 8. An informationprocessing method, comprising: in an information processing device thatincludes a central processing unit (CPU): designating, by the CPU, afirst position outside of a display region of an image display screen;setting, by the CPU, a trajectory of an augmented reality (AR) object,wherein the trajectory of the AR object is set as a plurality ofpositions; and controlling, by the CPU, the AR object, based on a secondposition of the display region, a third position of the AR object, auser's position, movement of the AR object towards the designated firstposition, and a height of the AR object, wherein the AR object ispassable through the display region of the image display screen.
 9. Theinformation processing method according to claim 8, further comprisingmatching, by the CPU, the height of the AR object with a central heightof the display region.
 10. The information processing method accordingto claim 8, wherein the information processing device is a glasses-typedevice having a pair of image display screens for a left eye and a righteye.
 11. The information processing method according to claim 8, furthercomprising arbitrarily setting, by the CPU, the height of the AR objectindependent of a central height of the display region.
 12. Theinformation processing method according to claim 8, further comprisingdisplaying, by the CPU, an image of the AR object based on an overlapbetween the AR object and the display region of the image displayscreen.
 13. The information processing method according to claim 8,further comprising displaying, by the CPU, the AR object on the displayregion to indicate a direction of a target object to a user.
 14. Theinformation processing method according to claim 8, further comprisingcontrolling, by the CPU, the movement of the AR object such that, animage of the AR object is displayed in the display region.
 15. Anon-transitory computer-readable medium having stored thereoncomputer-executable instructions that, when executed by a processor ofan information processing device, cause the processor to executeoperations, the operations comprising: designating a first positionoutside of a display region of an image display screen; setting atrajectory of an augmented reality (AR) object, wherein the trajectoryof the AR object is set as a plurality of positions; and controlling theAR object, based on a second position of the display region, a thirdposition of the AR object, a user's position, movement of the AR objecttowards the designated first position, and a height of the AR object,wherein the AR object is passable through the display region of theimage display screen.
 16. The non-transitory computer-readable mediumaccording to claim 15, wherein the operations further comprisingmatching the height of the AR object with a central height of thedisplay region.
 17. The non-transitory computer-readable mediumaccording to claim 15, wherein the operations further comprisingarbitrarily setting the height of the AR object independent of a centralheight of the display region.
 18. The non-transitory computer-readablemedium according to claim 15, wherein the operations further comprisingdisplaying an image of the AR object based on an overlap between the ARobject and the display region of the image display screen.
 19. Thenon-transitory computer-readable medium according to claim 15, whereinthe operations further comprising displaying the AR object on thedisplay region to indicate a direction of a target object to a user. 20.The non-transitory computer-readable medium according to claim 15,wherein the operations further comprising controlling the movement ofthe AR object such that, an image of the AR object is displayed in thedisplay region.